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Overview Mellon 


. _ University 
= UL 4600 standard for AV safety cases 

e Fully autonomous vehicles 

e Issued April 2020 


m Key 4600 ideas: 
e System-level safety case provides direction 
e Vehicle as well as infrastructure and lifecycle processes all matter 
e Safety metrics used for feedback loops 
e Third party component interface protects proprietary info 
e 4600 helps you know that you've done enough work on safety 
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= Traditional safety standards are prescriptive 


e “Here is how to do safety” (process, work products) 
— ISO 26262, ISO/PAS 21448, IEC 61508, MIL-STD 882, etc. 


= UL 4600 is goal based 


e “Here is what a safety case should address” 
— Do NOT prescribe any particular engineering approach 
» Use other safety standards within the safety case context 
e Standard for how to assess a safety case 
— Minimum coverage requirement (what goes in the safety case?) 
— Properties of a well-formed safety case 
— Objective assessment criteria 
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Example 4600 Clause Mellon 
University 

12.3.1 V&V shall provide acceptable coverage of safety related faults associated with the design phase. 
12.3.1.1 MANDATORY: 

a) Systematic design defects 

b) Design consideration of faults, corruption, data loss, and integrity loss in sensor data 

c) Requirement gaps/omissions and requirement defects 

d) Response to violation of requirement assumptions 

EXAMPLE: Response to exceptional operational environment 

e) Identification and description of the intended ODD 

f) Acceptable mitigation of aspects of the defined fault model for each component and other aspect of the item 
12.3.1.2 REQUIRED: 

a) Maintenance procedure definitions 

NOTE: While maintenance occurs during the lifecycle, the definition of procedures needs to correspond to design 

requirements and assumptions made in design regarding maintenance. 

b) Operational procedure definitions (including startup and shutdown) and operational modes 

c) Faults, corruption, data loss, and integrity loss in data from external sources 

d) Faults and failures associated with exceptional conditions that impair risk reduction functionality 

e) Hardware and software errata and other third-party component design defects 

f) Other faults in safety related functions, component designs, and other designed properties 
12.3.1.3 HIGHLY RECOMMENDED -N/A 
12.3.1.4 RECOMMENDED —N/A 


ian 4 


Carnegie 


Flexible Approaches Mellon 
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6.4.1 Each identified hazard shall be given a criticality level and assigned an initial risk assuming the 
absence of mitigation. 
6.4.1.1 MANDATORY: 
a) Hazard Log records criticality level and initial risk for each hazard 
6.4.1.2 REQUIRED: 
a) Use of at least one of the following risk evaluation approaches: 
1) Risk table 
2) Risk equation (weighted probability times severity) 
3) Fault Tree Analysis (FTA) 
4) Event Tree Analysis (ETA) 
5) Preliminary Item Safety Assessment(PSSA) 
6) Hazard Analysis and Risk Assessment (HARA) 
7) Bowtie diagram 
8) System-Theoretic Accident Model and Processes (STAMP) 
9) Field engineering feedback 
10) Other relevant risk evaluationapproaches 
b) Use of integrity level and related techniques 
EXAMPLES: Integrity level and related techniques from ISO 26262, IEC 61508; development assurance level from DO-178 


6.4.1.3 HIGHLY RECOMMENDED: 
a) Use of integrity levels defined in an accepted domain-relevant functional safety standard 
NOTE: It might not be practical to use such integrity levels for all aspects of an autonomous systems, but it is highly 
recommended to do so to the extent reasonable. 
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= Claim — a property of the system 
e “System avoids pedestrians” 

= Argument — why this is true 
e “Detect & maneuver to avoid” 

= Evidence — supports argument 
e Tests, analysis, simulations, ... 

= Sub-claims/arguments address 
complexity 
e “Detects pedestrians” // evidence 
e “Maneuvers around detected pedestrians” // evidence 
e “Stops if can’t maneuver” // evidence 


ARGUMENT 1 


EVIDENCE 1 


ARGUMENT 2 


Sub-CLAIM 2A 
~) 
Sub-ARGUMENT 2A 


EVIDENCE 2A 


Sub-CLAIM 2B 
) 
Sub-ARGUMENT 2B 


EVIDENCE 2B 
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= Everything needed to independently assess safety 
e Hazards and mitigation approaches 
e Claims traced: arguments to evidence 


m= Scope includes: Baal 
Technology: HW/SW, machine learning, tale, 

Lifecycle: deployment, operation, incidents, uinicnce: 2. 
Infrastructure: vehicle, roads, data networks, cloud computing, ... 
Road users: pedestrians, light mobility, emergency responders, ... 
Environment: Operational Design Domain (ODD) definition 

. and more ... 
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Example ODD Prompts (§8.2.2) an 
Behavioral rules Nil Ir 
e EXAMPLES: Traffic laws, vehicle path conflict resolution ie = eee aie 


priority, local customs, justifiable rule breaking for safety 


4 
= 


Compliance strategy of traffic rules and regulations C fur 7 -\ fF 
e EXAMPLE: Enumeration of applicable traffic regulations and mae & Fe | 3 
corresponding ego vehicle behavioral constraints https://bit.ly/2IKIZJ9 


Vulnerable populations including number, density, and types 
e EXAMPLES: Pedestrians, motorcycles, bikes, scooters, other vulnerable road users, other road users 


Special road user rules, if applicable 


e EXAMPLES: Bicycles, motorcycles, lane splitting, interacting with construction vehicles, oversize 
vehicles, snowplows, sand/salt trucks, emergency response vehicles, street sweepers, horse-drawn 
vehicles 


Seasonal effects 


e EXAMPLES: Foliage changes (e. g., leaves (dis) appearing), sun angle changes, seasonal behavioral 
patterns (e. g., summer beach traffic), seasonally-linked events (Oktoberfest, regatta crowds, fireworks 


gatherings, air shows) 
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= Safety Performance Indicator (SPI) 
e Like a KPI, but specific to safety 
e Provides metrics on safety case validity 


= SPI measures: 


e Behavior metrics for safety-related behaviors 
— E.g.: Acceptable violation rate of standoff to pedestrians 
e Assumption validity within safety case 
— E.g.: Tolerates gaps of up to X meters in lane markings 
— E.g.: Correlated camera and lidar false negative rate 
e Any other metrics that validate safety case 
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Feedback Loops Mellon 


University 


= Rather than assume perfection... 
.. Manage & improve imperfections 
e Feedback data incorporated in safety case 
e Convert “unknowns” into “knowns” over time 


[Edge Case Research] 


m Feedback loops for continuous improvement 
e Implementation faults 
e Design faults 
e Gaps in simulations, analysis tools, ... 
e Gaps in Operational Design Domain 
e Gaps in machine learning training data 
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= Reused or 3" party system “component” 
e Similar in spirit to ISO 26262 SEooC 
e Hardware, software, sensor, map data, ... 


m EooC has a safety case fragment 
e Vendor need not expose that safety case 
e Instead, provides an interface containing: 
— Properties &characteristics 2H —He 
- Assumptions that system must honor _[sw»-arcument2a] 4°" Te 
— Fault model used for assessment 


- 4600 clause coverage (might be partial) 
—- Assessment report 


ARGUMENT 2 


Sub-ARGUMENT 2B 


©) 
EVIDENCE 2B 
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Complementing Other Standards pe 


= ISO 26262, MIL-STD 882, etc.: potential starting points 
e Still useful where applicable 


= ISO/PAS 21448 etc. for scenarios 
e Design and validation process framework 
e SaFAD and emerging standards 


= 4600 has #DidYouThinkofThat? lists 
e Initial safety case coverage 
e Learn from experience: yours; others 
e Objective assessment criteria for safety case 
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Other Key Points Mellon 
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= Self-certification is permitted 
e Internal assessor permitted; no external “certificate” requirement 
m Only necessary technical mitigations required 


e “Does not apply to this system” and “Outside ODD” are OK wh a609 
e Can use non-technical mitigations So stavan eget 
m Underwriters Laboratories is a non-profit SDO s idicwionabattensitae CAG 


e Voting committee (STP) has diverse representation 
e Continuous Maintenance process provides timely updates 
= Does 4600 conflict with ISO 26262 or ISO/PAS 21448? 
e No 
m What if you can't afford to buy a copy? 


e Issued standard is free to browse (“digital view”) on-line in its entirety: 
https://www.shopulstandards.com/ProductDetail.aspx?productid=UL4600 
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UL 4600 Version 2 Ur sity 
= Issued March 15, 2022 ANSI/UL 4600 2™ Edition 
a Evaluation of Autonomous Products 
m Assessment terminology & roles: Gh) usin 
e Self-assessment dented 


Edition Date: March 15, 2022 


— Development team vets safety case 
e Independent assessment 
— Scope includes independent technical substance of safety case 


= Safety case terminology and structure 
e Significant improvements; same ideas and intent as version 1 
= Terminology 
e Improved alignment with other standards 
= Other improvements per stakeholder feedback —gsosspitipkoopman 14 


ANSI Approved: March 15, 2022 
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UL 4600 Version 3 - In Progress for 2022 —_—_Pitlon 
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= Primary goal: specific coverage of heavy trucks 
e Expands scope, but no fundamental change was required 
m Revised safety case framework for autonomous trucking 
e Adds concept of platoon (coordinated vehicles with a safety buffer) 
e Various related added prompts (e.g., hazardous materials) 
m Revised to add examples specific to autonomous trucking 
e Cargo loading/unloading operations 
e Communication with trailing platoon vehicles 
= Other improvements 
e Added a preferred Safety Performance Indicator approach 
e Emergency responder terminology 
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Review of Key Ideas Mellon 
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= System-level safety case provides direction 
e Highlights gaps in evidence and arguments 
= Vehicle, infrastructure, and lifecycle processes all matter 
e If safety case depends upon it, that makes it safety related 
= Metrics combine with feedback loops 
e Operational feedback will be essential for practical safety 
= Third party component interface to protect proprietary info 
e EooC interface permits separate component assessment 
= 4600 helps you know that you've done enough safety work 
e Robust prompts and pitfalls capture best practice/lessons learned 
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